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Abstract 


This study aims to examine the feasibility of a number of technical solutions implemented in a web-based system designed for 
the creation and management of online language exams within PAULEX Universitas, a project for the development of an online 
platform to design, deliver and assess the foreign language exam within the Spanish national University Entrance Examination. 
The paper provides an overview of the context and a discussion of current changes in Spanish ministerial policies, followed by a 
description of the PAULEX testing platform and a discussion on the technological solutions adopted. Lastly, the paper reflects 
upon a quantitative analysis derived from a pilot experience conducted in Valencia with over 200 students and their respective 
teachers, the results of which indicate that in spite of the difficulties, the implemented solutions have a high degree of acceptance. 
© 201 1 Published by Elsevier Ltd. 
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1. Introduction 

The PAULEX Platform is a multilingual content-management system for language learning and assessment 
which includes an authoring tool, a content manager and an online assessment environment. The platform was 
created within the PAULEX-Universitas Project, carried out by the CAMILLE research group at the Universidad 
Politecnica de Valencia. The project is now in its piloting stage of the platform and the analysis of the results 
deriving from the different trials that have been conducted. Some of the findings concerning the testing and 
validation phase will be mentioned in section 3 below. 

The PAULEX platform is based on the InGenio e-leaming platform, also developed by the CAMILLE research 
group (Gimeno 2005, de Siqueira et. al. 2009) and therefore, it shares some of its features. PAULEX caters for the 
development and management of Internet-based assessment while fulfilling the security, scalability and usability 
requirements to simultaneously evaluate a large number of students in high-stakes selective processes, such as the 
Spanish University Entrance Examination. Its online format allows the information to be constantly updated and 
stored in the central servers, and the users access the system through Internet security applications such as the use of 
passwords, IP authentication and the HTTPS protocol. The platform features an authoring tool to develop and 
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publish the web-enabled language exams, as well as providing an online student assessment and management 
environment (Gimeno 2008). As for its users, they fall into at least one of the four following categories: content 
providers, students, markers and administrators. 

2. The PAULEX platform 

As mentioned above, there are four categories concerning the users who can access the PAULEX Platform. 
These users can be i) content providers, i.e. the test or exam writers, who design and develop the exercises and tasks 
included in the exams by means of the authoring tool; ii) students, who register and log-on to the delivery platform 
to gain direct access to their exam; iii) markers, who have access to the assessment module, through which they 
visualise the exercises or tasks to be assessed complying with the following criteria: confidentiality and protection of 
the data contained in each exam; and iv) administrators, who use the administrator(s) module to manage the 
platform as well as all the exam results, since they have permission to manage users and their corresponding profiles 
and tasks while administering the exams and the exam results. Each user is only allowed to manage their own 
activities within the system and for the sake of reliability, only the administrators have access to the main data, e.g. 
the management of an exam schedule or access to the profile of a student in particular, should this information be 
required in a revision process. 

Once the users are registered on the system and depending on the permissions given to them by the system 
administrator, the user or users who take the role of content provider(s)/ test designers(s) can create and manage 
whole exams, specific tests and/or individual exercises. An author can only access the materials created by other 
authors under the administrator's permission. For example, markers are the users in charge of monitoring the 
students assigned to them or the students registered on a particular course, so they would have full access only to the 
activities completed by these students. To safeguard anonymity, markers access the exams unnamed and the 
identities of the students are granted confidential according to the criteria previously established by the 
administrator. As for the administrator, this user can edit personal data and give permissions to the rest of the users 
(materials writers, exam markers and students), and they are also responsible for the management of tasks or exams 
within the platform as well as for adjusting the appropriate system settings in relation to the processes of creation, 
implementation, marking and analysis of the exam results. Figure 1 illustrates the vision of an administrator in 
PAULEX, highlighting the fact that for every section of the main menu there is a system control to award 
permissions. Thus, every user is only allowed to access the information and functions they are authorised to access. 



The higher the flow of information, all the more necessary it becomes to ensure information control and 
protection, both of which can be provided through different data protection systems such as user authentication and 
encryption (Kent 1998; Liao 2005; Yang 2006). In the data control process, IT (information technology) systems 
combine different kinds of technologies, the degree of refinement of which can vary considerably according to the 
technical resources available in each educational setting, both in terms of hardware and software. Another factor to 
be taken into account is the legislation in force in the regions where these systems are to be used. For example in 
Spain, the use of security systems such as the electronic signature are regulated by law (59/2003 Law - 19 December 
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2003). Additionally, in order for the system to be suitable for use in different educational locations, PAULEX has 
intentionally been developed to run with minimal software requirements and to be used on computers with varying 
software and hardware features. To this end, the system avoids the installation of plug-ins and additional 
programmes. It also runs on any common operating system (Windows, Linux, Macintosh and Unix). The only 
requirements are a live Internet connection and a media player application to play audio and video and to record the 
students’ utterances. The platform is controlled by a central server (CS) and a back-up server. The back-up server 
automatically duplicates all the information on the CS should it be necessary to replace or work on it to repair any of 
its components. PAULEX is basically divided into two main parts: the database and the source code. The database is 
based on PostgreSQL, one of the most efficient open source database management systems available on the Internet. 
The multimedia files saved on the server are also part of the PAULEX database, a feature which allows 
administrators to build an abundant pool of images and audio-visual files, including the audio files recorded by the 
students when completing the oral section of the exam through voice-recording systems. 

3. The validation phase 

The PAULEX project emerged as an attempt to design and to test the feasibility of implementing a computer- 
assisted English exam for the official University Entrance Examination in the Valencian Community, an 
examination which is managed by the department of education of the regional government in collaboration with the 
five public universities located in its territory. The need arose as a the result of the newly approved national 
legislation (Royal Decree 1892/2008) which stated that, as from the year 2012, the foreign language exam should 
not only include assessment of reading and writing skills, but also listening and speaking skills. The CAMILLE 
research group therefore set about designing an online assessment environment that would be accepted by students 
and teachers alike, as well as complying with the new government regulations. 

In this process, one of the first steps taken was to find out what the most accepted items were in order to define 
the most appropriate construct for the exam (Garcia-Laborda, 2010). The study was conducted with 214 secondary 
school English teachers who prepare the students for the entry exam from the three Valencian provinces (Alicante, 
Castellon and Valencia) by means of a questionnaire focusing on their preferred exercise typologies for each of the 
four skills to be assessed. The main findings drawn at that stage concluded that teachers chose the activities which 
most resembled the current pen and paper exam. This verified the fact that, in spite of their eagerness to change the 
format of the exam and update it to current trends, their preferences were in line with the more traditional options 
and they conveyed their reluctance in terms of using computers in their regular English language classes in 
preparation for the entrance examination. The full results of the study have been published in Martinez, Sevilla and 
Gimeno (2009). 

Taking these results into consideration, the team then designed an online English exam delivered through the 
PAULEX platform that students took during the first pilot trial. The exam included assessment of all four skills, i.e. 
reading, writing, listening and speaking. Just over 200 students in their final year at secondary school participated in 
this first trial which was carried out in computer labs both at several Valencian schools and at the University. The 
computer engineers had to do minor adjustments to adapt all the computers to the platform’s specific requirements. 
For instance, programmes such as Realplayer were installed so that the audio and video files could be played. In 
general, the results have been very satisfactory. The students who took the online exam felt that the platform is user- 
friendly, especially when editing their answers, watching the video files and speaking into the microphone to record 
their voice. One of the drawbacks was precisely the students’ shyness when speaking into a microphone and 
recording their own voices after being prompted to do so, although this, we think will be overcome if the online 
platform is adopted by schools and learners are given more opportunities to practice on the online platform. Once 
the audio file was recorded, the student could listen to the recorded file as many times as he or she wished, as well 
as being able to delete and rerecord it. 
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Figure 2. Sample listening comprehension exercise. 


There were, however, time constraints as students could only devote their English lesson and break time to doing 
the exam. They only had 60 minutes to complete it, instead of the stipulated 90 minutes they would normally be 
allowed if sitting the actual exam. In spite of this, in most cases, students did not need to exceed these 60 minutes to 
complete the exam, which included the newly incorporated listening and speaking sections. Students were requested 
to complete all the exam exercises and validate their answers and any modifications made thereafter, and then 
electronically send them once they had finished for storage on the server. In the listening section, they had to watch 
a number of short video sequences and then they had to listen to the questions and write or select their answers. The 
speaking part was the most interesting one for research purposes. The students were asked to take a look at an eye- 
catching image showing logos of social networks and online chat programmes. They were given four minutes to 
record their voices analysing the image provided and relating it to their own personal likes and experiences. Many of 
the students devoted some time to writing a script that could allow them to produce a more structured speech. 

Once students had finished the exam, validated their answers and sent them to the server, a new window 
displayed a questionnaire through which students were encouraged to report their personal experience using 
technology and their attitude toward doing a computer-assisted exam. The results showed that they felt even more at 
ease with this format than with the traditional pen and paper one because they are accustomed to using computers 
and the widespread resources available on the Internet. The secondary school teachers who witnessed the whole 
process expressed their positive opinions and impressions as they had expected a higher level of anxiety on their 
students’ behalf because of the new exam format and because of the inclusion of sections devoted to testing oral 
skills for the first time ever in a university entrance exam in Spain. 

4. Conclusion 

The results obtained through observation and through the questionnaires indicate that both the students who have 
faced the computer-assisted test and the teachers who have witnessed this process have a very positive attitude 
towards this mode of learner assessment. They have expressed their enthusiasm and have felt much less disinclined 
both to incorporating the evaluation of oral skills to the foreign language exam within the university entrance 
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examination and to doing the exam on a computer. The conclusion after having tested the system in real conditions 
is that teachers and students seem to be in favour of this proposal. The trials that have been conducted have also 
shown that implementing a computer-assisted English exam as part of the Spanish University Entrance Examination 
is feasible. The exam results achieved by the students have also been very satisfactory, although teachers and 
students were advised that the marks obtained in the pilot trial would not necessarily be equivalent to the ones that 
they would be awarded when facing the real exam. Most of the exercises were assessed automatically by the system, 
although personalised marking was required for the sections where students had to answer open-input questions in 
writing and record their answers on the media player. These exercise types are assessed by having the exam markers 
access a specially designed module enabling them to anonymously mark the students’ written and oral production 
for which they are provided with a set of pre-established criteria and scoring system to apply. 

To conclude, the PAULEX platform is a robust and dynamic tool enabling the administrators of the Spanish 
official University Entrance Examination to deliver the foreign language exam paper via the Internet. Several years 
of research were needed in order to be able to present a solid and feasible proposal for the computerisation of the 
exam, which has emerged as one of the most achievable alternatives in facing the challenges arising from the 
changes brought about by the new Spanish educational legislation. 
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